Coben, D., Millcr-Reilly, B., Satherley, P., & Earle, D. (2016). Making the most of PIAAC: Preliminary 
investigation of adults’ numeracy practices through secondary analysis of the PIAAC dataset. Adults 
Learning Mathematics: An International Journal, 1 7(2), 27-40 


Making the most of PIAAC: 

Preliminary investigation of adults’ numeracy practices through 
secondary analysis of the PIAAC dataset 


Diana Coben 

University of Waikato, New Zealand 
<dccoben@waikato. ac .nz> 


Barbara Miller-Reilly 


University of Auckland, New Zealand 
<b .miller-reilly@ aukckland. ac. nz> 


Paul Satherley and David Earle 

Ministry of Education, New Zealand 
<paul.satherley@minedu.govt.nz><david.earle@minudu.govt.nz> 


Abstract 


The Programme for the International Assessment of Adult Competencies (PIAAC) assesses key 
information processing skills and collects information on how often people undertake a range of activities 
at work and in everyday life. We are exploring what secondary analysis of online anonymised PIAAC data 
can tell us about adults’ numeracy practices. In the process we review the accessibility and user- 
friendliness of the data for novice researchers and practitioners in the hope of encouraging them to explore 
this rich resource and give a brief account of our experience of the process of accessing publicly-available 
PIAAC data for secondary analysis. 
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In this paper we explore what secondary analysis of data from the latest international Survey of 
Adult Skills in the Programme for the International Assessment of Adult Skills (PIAAC 1 ), can 
tell us about adults’ numeracy practices. In the process we are also reviewing the accessibility 
and user-friendliness of the dataset. We hope to encourage the exploration of this rich 
resource by practitioners and researchers, including those with little previous experience of 
working with large datasets. This is important because, as Hansen and Vignoles(2007, p. 1) point 
out “In the last few decades, there has been an unprecedented increase in the availability and 
quality of large-scale data sets that are suitable for use in education research. Analyses of these 
data have the potential to radically improve the robustness and generalisability of educational 
research”. In a still young but growing field such as adults learning mathematics, this is 
especially important. 

We focus on secondary analysis of anonymised publicly-available PIAAC data. We draw our 
understanding of secondary analysis from Dale, Watham and Higgins (2008, p. 520): 
“Secondary analysis is generally understood as the analysis of data originally collected and 
analysed for another purpose”. In addition, Heaton (1998) says “Secondary analysis involves the 
use of existing data, collected for the purposes of a prior study, in order to pursue a research 


’http://www. oecd.org/site/piaac/surveyo fadultskills.htm. 
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interest which is distinct from that of the original work; this may be a new research question or 
an alternative perspective on the original question”. She adds that secondary analysis may be 
undertaken either by the original researchers or others. 

The OECD has published two analytical reports on PIAAC (OECD, 2013a, 2016a). They relate 
to the first and second rounds of the PIAAC survey in participating countries. The New Zealand 
Ministries of Education, and Business, Innovation and Employment have published three initial 
reports: Ministry of Education & Ministry of Business, Innovation and Employment (2016a, 
2016b, 2016c). Secondary PIAAC analysis will tackle research questions and topics beyond 
what is covered in the initial reporting, or take different or more in-depth approaches. Examples 
include: 


• integrating different parts of the PIAAC dataset so as to generate new knowledge and 
understanding of associations and relationships 

• building or refining conceptual or statistical models. 

• exploring specific themes rather than taking a broad focus. 

This paper explores an example of secondary analysis of numeracy practices in everyday life and 
work that takes a different perspective from the analysis of OECD (OECD, 2016a, pp. 97-113) 
which constructs indices that group together tasks involving similar activities. 


What is PIAAC? 

PIAAC is an international survey that assesses key information processing skills of adults of 
working age 2 in Literacy (reading) and Numeracy and collects information on how often they 
undertake a range of related activities in work and everyday life (OECD, 2013c). Two additional 
assessment components are optional for participating countries: Reading Components (Sabatini 
& Bruce, 2009); and Problem Solving in Techno logy-rich Environments (OECD, nd-b). PIAAC 
builds on previous international surveys: the 1994-1998 International Adult Literacy Survey 
(IALS) (OECD & Statistics Canada, 2000);and the 2003-2006 Adult Literacy and Life Skills 
Survey (ALL) (Satherley, Lawes, & Sok, 2008; Statistics Canada & OECD, 2005). 

PIAAC is planned as a repeating survey with a ten-year cycle. The first cycle was undertaken in 
two rounds, with a third round scheduled for 2016-2019. Round One included 24 countries 3 with 
data collected in 2011/12 and findings published in 2013 (OECD, 2013a). New Zealand, together 
with eight other countries 4 , participated in Round Two, with data collection April 2014 - 
February 2015. Results were released in June 2016 (OECD, 2016a). 

Some key New Zealand findings are: 

• New Zealand adults’ literacy and problem solving skills are on average among the 
highest in the OECD 

• New Zealand adults’ numeracy skills are on average higher than the OECD average 

• Although there are significant differences in skills between ethnic groups, average 
literacy and numeracy skills have been rising faster among Maori and Pasifika than in 
the total New Zealand population 


2 Normally 16-65 years but this can vary, e.g., Australia extended the range to 15-74 years. 

3 Round 1 PIAAC countries were: Australia, Austria, Belgium, Canada, Cyprus, Czech Republic, Denmark, Estonia, 
Finland, France, Germany, Ireland, Italy, Japan, Korea, The Netherlands, Norway, Poland, Russian Federation, Slovak 
Republic, Spain, Sweden, United Kingdom, the United States. 

4 Round 2 PIAAC countries were: Chile, Greece, Indonesia, Israel, Lithuania, New Zealand, Singapore, Slovenia, and 
Turkey. Round 3 PIAAC countries will be: Argentina, Colombia, Hungary, Kazakhstan, Mexico, Peru and the United 
States of America (the latter repeating PIAAC in subpopulations). 
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• Overseas-bom New Zealanders have on average higher literacy and numeracy scores 
than overseas-born people in any other country 

• While there are no differences in average literacy and problem solving skills between 
men and women, men have higher numeracy skills on average than women. 

The PIAAC survey is carried out by: interviewing a sample of at least 5000 adults in each 
participating country 5 in their homes; collecting a broad range of information through a 
Background Questionnaire 6 ; and assessing skills in the PIAAC domains. The language of 
assessment is normally the official language or languages of each participating country 7 . 
Depending on their computer skills 8 , participants either enter their responses to the assessment 
items of the main skill domains on the assessor’s laptop computer or complete a paper version 
using printed test booklets. The Background Questionnaire is administered face-to-face in the 
respondent’s home by an interviewer who enters the answers into a laptop computer. All aspects 
of countries’ implementation of PIAAC is strictly monitored and quality-assured by the PIAAC 
Consortium. PIAAC governance is provided by a Board of Participating Countries. 

The Background Questionnaire collects data on participants’ educational background, skills used 
at work (for those currently or recently in employment) and in other contexts such as the home 
and the community. For example, people are asked about their voting habits, volunteering, 
languages spoken, political efficacy and health. The following variables are covered: 
demographic characteristics; other personal characteristics (including learning disposition and 
self-assessed health status); education and training characteristics; work characteristics; self- 
assessed mathematics skills for work; self-assessed reading and writing skills for work; and skill 
use in everyday life. 

Respondents with very low literacy skills (as assessed by some initial questions) are not assessed 
in the main skill domains, but instead go directly to the Reading Components assessment. This 
covers “the basic set of decoding skills that enable individuals to extract meaning from written 
texts: knowledge of vocabulary, ability to process meaning at the level of the sentence, and 
fluency in reading passages of text” (OECD, nd-a, p. 1). 

In PIAAC, proficiency is considered as a continuum of ability involving information-processing 
tasks of increasing complexity defined as ‘proficiency levels’ (OECD, 2013b, p. 64). Six 
proficiency levels are described for Literacy and Numeracy (Levels 1 to 5, and below Level 1) 
and four for Problem Solving in Technology-rich Environments (Levels 1 to 3, and below Level 
1). These summarise what adults with proficiency scores in each skill domain can do. The 
‘average’ individual with a proficiency score in the range defining a level will successfully 
complete items located at that level approximately two-thirds of the time. PIAAC measures 
cognitive skills through test items that have a range of contexts: work-related, personal, society 
and community, and education and training. The OECD has released a number of sample test 
items (OECD, 2016b, pp. 21-23; 26-28; 31-32). The work context of PIAAC’s cognitive skill 
assessment is complemented by self-reports on generic skills required in work, including inter¬ 
personal skills and physical skills. 

As for earlier international skill surveys, PIAAC published results include: average scores on 
each skill domain for countries and population subgroups; comparisons over time and with other 
countries; and proportions of different sub-populations reaching different benchmarks for each 
skill domain. 


^Participating countries can increase the sample size to provide better information on specific sub-populations; e.g., 
New Zealand oversampled 16-25 year olds and Maori. 

6 The PIAAC Background Questionnaire is at http://www.oecd.org/site/piaac/BQ_MASTER.HTM. 

7 In some countries, PIAAC assessment has also been conducted in widely-spoken minority or regional languages. 

8 Problem Solving in Technology-rich Environments is assessed only on a computer, so only participants who have 
sufficient computer skills and choose to use a computer are assessed on this domain. 
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PIAAC breaks new ground by: expanding the range of skill domains measured; including 
information on the skills of adults with levels of proficiency below Level 1; expanding the self- 
reported measures of the use of skills at work; introducing self-reported measures of 
qualifications matched to work; using computers to administer this kind of international 
assessment; making data publicly available for review and secondary analysis on an 
unprecedented scale via the PIAAC website 9 ; having an online version of the assessment 
publicly available 10 ; and not specifying Level 3 as a benchmark (OECD, 2010, p. 4).In IALS, 
Level 3 was considered to be the minimum skill level required to cope with the demands of 
modem society (OECD & Statistics Canada, 2000, p. xi). However, while many higher-level 
jobs require Level 3 or above, there is no evidence that everyone needs to be at Level 3. This 
change is significant because, as Black and Yasukawa (2014) point out, the Level 3 criterion has 
been used by powerful institutions to promote a crisis discourse in adult literacy and numeracy. 

For more detail about PIAAC constructs, methods and how the survey was undertaken in 
participating countries, see the PIAAC Reader's Companion(OECD, 2016b). 


Numeracy in PIAAC 

Numeracy 11 is defined in PIAAC as: 

the ability to access, use, interpret and communicate mathematical information and 
ideas in order to engage in and manage the mathematical demands of a range of 
situations in adult life ... Numerate behaviour involves managing a situation or 
solving a problem in a real context, by responding to mathematical 
content/information/ideas represented in multiple ways. (OECD, 2012, p. 34) 

PIAAC directly measures numeracy proficiency through the Numeracy assessment. This 
provides average scores on the PIAAC Numeracy scale for the whole population, or for 
subgroups. 

PIAAC also collects numeracy-related information via the Background Questionnaire. The 
interviewer asks respondents how often they undertake seven 12 numeracy practices in work (if 
appropriate) 13 and everyday life. For example: 

The following questions are about activities that you undertake as part of your job 
and that involve numbers, quantities, numerical information, statistics or 
mathematics. 

• In your job, how often do you usually calculate prices, costs or budgets? 

• Use or calculate fractions, decimals or percentages? 

• Use a calculator - either hand-held or computer based? 

• Prepare charts, graphs or tables? 

• Use simple algebra or formulas? 

• Use more advanced mathematics or statistics such as calculus, complex 
algebra, trigonometry, or use of regression techniques? 


TittpV/www. oecd.org/site/piaac/publicdataandanalysis.htm 

10 A public online version of the PIAAC survey is at http://www.oecd.org/skills/ESonline-assessment/. 

u The Numeracy domain in the ALL survey is comparable with the Numeracy domain in PIAAC but not with the 

Quantitative Literacy domain in IALS. 

12 The international version of the PIAAC Background Questionnaire includes the first six of these numeracy activities. 
New Zealand asked for the seventh activity, ‘measure or estimate the size or weight of objects’, to be added as a 
national extension. This activity was in the 2006 ALL survey and it was found to be analytically useful as it helps 
identify groups of people whose numeracy activities have this simple practical purpose. 

13 The same questions are asked in the past tense for people who are not currently working, but who worked in the last 
12 months. 
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• Measure or estimate the size or weight of objects? 

The frequency options are: Never; Less than once a month; Less than once a week hut at least 
once a month; At least once a week but not every day; Every day. These data provide measures 
of the frequency and diversity of participants’ numeracy activities. The activities can be analysed 
separately, or a range of options can be developed for deriving an index of numeracy activity by 
combining the data across the activities. 

The Adult Literacy and Life Skills Survey (ALL) asked similar questions, and since analysis of 
these questions, together with occupational characteristics, provided a coherent picture, we have 
some good assurance of the validity of these measures (Satherley, Lawes, & Sok, 2009). 


Analysing adults’ numeracy practices: issues and types of analysis 

A range of types of analysis may be used to explore questions about adults’ numeracy 
practices 14 . Simple univariate tabulations can provide a ‘big picture’ view while multivariate 
analysis such as multiple regression can provide measures of the contributions of different 
factors to associations with numeracy practice. For example, frequent practice of numeracy 
activities at work may be associated with: higher levels of education; numeracy-related fields of 
study; specific groups of occupations; and higher measured numeracy skill. Such analysis can 
show which factors are most strongly associated with frequent numeracy practice and how much 
increase in frequency of numeracy practice is associated with one unit of measured numeracy 
skill, whilst holding other factors constant. 

PIAAC data cannot tell us to what extent frequent numeracy practicecauses high numeracy skill, 
so researchers should not use ambiguous language such as ‘leads to’, ‘brings about’, 
‘influences’, or ‘is li nk ed with’. In any case, it may be that numeracy practice, opportunity or 
requirement to undertake numeracy practice, and numeracy skill are all mutually reinforcing. 
Even where data show a strong association between two factors (for example, numeracy practice 
and numeracy skill) whilst controlling for other factors, we cannot infer that a change in one 
factor will result in a change in the other, either on an individual or a group level. Another 
limitation is that PIAAC does not provide a measure of either the intensity or complexity of 
numeracy activity. For example, finance analysts doing nothing but calculating costs and budgets 
would report the same way as someone who worked on costs and budgets for 10 minutes every 
day. Researchers also need to be aware of issues relating to continuous and categorical variables. 
For example, PIAAC frequency options are five separate categories, although for some 
analytical purposes it may be legitimate and useful to derive a continuous frequency variable 
from the discrete categories. 

Some other methodological or technical issues users should be aware of include: 

• The 2016 PIAAC report (OECD, 2016a) presents OECD averages based on the 28 
OECD countries participating in Round 1 or Round 2. This differs from the averages 
presented in (OECD, 2013a) based only on Round 1 countries. 

• Numeracy skill is directly measured, whereas participation in numeracy activities is self- 
reported. 

• The PIAAC data are rich enough to provide scope for different indicators of a construct, 
such as job-skill mismatch. What indicator is the most useful will depend on the purpose 
of the analysis, or the need for comparability with published findings across countries. 


14 We recommend novice researchers should consult a text book covering quantitative research methods in the social 
sciences (e.g., De Vaus, 2014; Frankfort-Nachmias, Nachmias, & DeWaard, 2015). 
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• Where a user analyses PIAAC data across all or many participating countries, the large 
sample may mean that nearly all differences are statistically significant. This will not 
apply for an analysis for one or a few countries, and very fine analysis with many 
variables for a single country may entail large sampling errors and therefore few 
significant differences. 

Some questions about numeracy activities that PIAAC secondary analysis can help answer 

In this section we pose some questions to demonstrate the process we went through in our 
exploration of the PIAAC dataset and comment on how these questions could be answered 
through secondary analysis of the PIAAC data. The OECD has published an initial analysis of 
skill use at work and in everyday life (OECD, 2016a, pp. 98-100). This is a country comparison 
that aggregates activities and summarises frequencies into an index. Our paper aims to suggest 
more in-depth and more detailed analysis focused on more specific research questions beyond 
that of the OECD report. 

How often do people perform numeracy activities at work and in everyday life? 

PIAAC numeracy activities at work and in everyday life and frequency categories can be shown 
in tables as simple frequency tables, i.e., activity categories can denote rows with frequency 
categories, denoting columns 15 . 

What are the patterns for different groups in numeracy activities at work? 

The PIAAC Background Questionnaire sheds light on the part numeracy activities play in 
different kinds of work for different groups of people. PIAAC classifies Occupations 16 as: 
Managers', Professionals', Technicians; Clerical', Service & sales; Agriculture & fisheries 
workers; Trades; Machine workers; and Labourers. Even at this broad level, analysis of the 2006 
New Zealand ALL data show distinct profiles of numeracy activity for different occupations, for 
example, managers, trades workers, technicians, professionals, and agriculture and fisheries 
workers engaged in relatively frequent numeracy practice compared to clerical, labourers, 
machine and service and sales workers (Satherley et al., 2009). Broad PIAAC Industry sector 
groups include: Agriculture & fisheries; Manufacturing; Construction; Trade; Transport & 
communications; Finance & real estate; and Health & education. Analysis of PIAAC data will 
enable researchers to identify sectors with unexpected pockets of high or low numeracy activity. 
We can also look at the association between numeracy practice and measured numeracy skill to 
answer questions such as: 

• How likely are people with strong numeracy skill to have jobs that entail frequent 
numeracy activity? 

• Where in the economy do we see areas of job-skill mismatch? One way of tackling this 
question is developing an indicator of job-skill match that compares actual numeracy skill 
with the frequency of undertaking numeracy activities. A possibility is grouping 
respondents into (a) high, medium and low numeracy skill and (b) high, medium and low 
frequency of undertaking numeracy activities at work; then derive a third grouping: over¬ 
skilled, under-skilled and matched; then we could investigate industry or occupation 
patterns for this indicator (see OECD, 2016a, pp. 129-143). 

Analysis by socio-demographic characteristics can also show different numeracy activity 
patterns for different groups by: gender; age group; education level; and field of study. New 
Zealand’s oversampling of 16-25 year olds and Maori will allow these groups’ numeracy 
activities to be examined in detail. 


15 See Cross Tabulation Analysis: https://www.qualtrics.com/wp-content/uploads/2013/05/Cross-Tabulation- 
Theory.pdf 

16 PIAAC dataset categories and sub-categories are italicised for ease of reference. 
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What are the patterns for different groups for numeracy activities in everyday life? 

We can analyse numeracy activities in everyday life similarly to work contexts. In this case 
analysis is likely to focus on socio-demographic characteristics and measured numeracy skill. 

What similarities or differences exist for numeracy activity at work and in everyday life? 

By looking at participants’ numeracy activities at work and in everyday life, we may discern 
patterns of global differences between these two contexts for individuals. For example, if we 
were interested in calculator use in work compared to in everyday life, we could cross-tabulate 
these variables. 

What similarities or differences exist for numeracy and literacy activity at work? 

Literacy activities at work included in the Background Questionnaire cover eight reading and 
four writing activities: Read directions or instructions; Read letters, memos or emails; Read 
articles in newspapers, magazines or newsletters; Read articles in professional journals or 
scholarly publications; Read books; Read manuals or reference materials; Read bills, invoices, 
bank statements or other financial statements; Read diagrams, maps or schematics; Write 
letters, memos or emails; Write articles for newspapers, magazines or newsletters; Write 
reports; and Fill in forms. One approach to making sense of this level of detail is to summarise 
the numeracy activities by creating a frequency index. For example, we could assign a numerical 
code for the frequency options and add the codes to obtain a total score, which would be a 
measure of the frequency and diversity of numeracy activities undertaken. Someone with a high 
score often engages in several different numeracy activities. We could generate similar indices 
for reading and/or writing. This would allow us to study the characteristics of, for example, 
people who score highly on both numeracy and literacy activities at work. 

What similarities or differences exist for numeracy and literacy activity in everyday life? 

We could take a similar approach to explore numeracy and literacy activity in everyday life. For 
example, we could look at questions about whether numeracy and literacy activities in everyday 
life go together for many people or not. What are the characteristics associated with undertaking 
a lot of different numeracy and literacy activities? Can we identify characteristics that are 
associated with seldom undertaking few numeracy and literacy activities? 

What can PIAAC tell us about adults’ financial capability? 

PIAAC can shed light on financial capability, for example, through a study of the relationships 
between directly measured numeracy skill and the self-reported numeracy activity of calculating 
prices, costs or budgets. Similarly we could study literacy skill and the literacy activity of 
reading bills, invoices, bank statements or other financial statements. Acknowledging that 
numeracy activity is self-reported, inferences about financial literacy skill may still be possible. 

What are the patterns of work-related numeracy activity together with ICT activities? 

The PIAAC Background Questionnaire asks how often people participate in the following ICT 
activities: use email; use the internet in order to better understand issues related to your work; 
conduct transactions on the internet; use spreadsheet software; use a word processor; use a 
programming language to program or write computer code; participate in real-time discussions 
on the internet. These could be analysed using a similar approach to the comparisons between 
numeracy and literacy activities. 

What are the characteristics of people who seem to be matched or mismatched on 
numeracy skill at work and numeracy activity? Are Field of study of highest qualification, 
or Occupation related to the match or mismatch? 

To what extent people’s numeracy skill aligns with their numeracy activity is an important issue. 
A working-age population might contain significant proportions who could easily manage more 
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(or a higher level of) numeracy activity than they actually engage in, or significant proportions 
who are attempting numeracy activity that their skills do not support. In these cases, a work 
skills policy issue seems to emerge about whether policy levers should be applied to support a 
more efficient allocation of numeracy skills and numeracy activity. 

One approach to better understanding numeracy skill and activity mismatch would be a 
multivariate analysis of Numeracy activity. Measured numeracy skill, and Main field of study of 
highest qualification, and Occupation. This could build on an OECD report on the first round of 
PIAAC entitled The System-level Causes and Consequences of Field-of-study Mismatch (Montt, 
2015). 

What changing patterns for young people’s numeracy activities can we see on their 
pathways from school to tertiary education to work? 

Here we could identify subgroups of 16-25 year olds who are: at school; in formal tertiary 
education; in full-time work; or combining work with study. We could identify what 
characteristics are strongly associated with participating in numeracy activities in everyday life 
(including study). These could include respondents’: age; gender; level of highest qualification; 
the level of qualification they are studying for; the main subject of qualification they are 
studying for; the type of educational institution they are studying at; whether they are also 
undertaking non-formal or informal study; and their occupation and numeracy activities at work 
(for those currently or recently working). 

How have patterns of numeracy activity at work changed over time? 

The ALL survey asked how often people participated in six numeracy activities as part of their 
job. The ALL activities were: Measure or estimate the size or weight of objects; Calculate 
prices, costs or budgets; Count or read numbers to keep track of things; Manage time or prepare 
timetables; Give or follow directions or use maps or street directories; and Use statistical data 
to reach conclusions. The first two on the above list of activities were re-asked, the first in New 
Zealand’s PIAAC survey only, and the second internationally. Using these data, we could look at 
how often workers undertook the numeracy activities in common between ALL and PIAAC, and 
investigate whether changes over time are associated, for example, with changing work, with 
changing occupational composition, or changing levels of education. 

We turn now to a brief account of our experience of the process of accessing publicly-available 
PIAAC data for secondary analysis. 


Using PIAAC Data Explorer to access PIAAC data 

Information on PIAAC Public Data & Analysis is available online 17 . The OECD PIAAC 
Gateway website Data Tools > Datasets and Tools 18 is designed to give users the tools needed to 
analyse the PIAAC dataset. The Data Tools section provides information on how to analyse a 
chosen dataset and Data Files are available, categorised as National, International, and Trend. 
Users are advised to read the guide What You Need to Consider{AlR PIAAC Team, nd) and 
watch online Distance Learning Data Training (DLDT) modules 19 to learn about appropriate 
statistical procedures and methods of analysis before accessing the PIAAC datasets. Users may 
also wish to consult a survey statistician or a psychometric expert for technical advice. 

The PIAAC International Data Explorer (IDE) 20 is a web-based application for accessing PIAAC 
data that does not require any advanced statistical knowledge or specialist software. It is a point- 
and-click interface for creating statistical tables and charts and exploring levels of adult skills 
and demographics. 


17 http://www.oecd.org/site/piaac/publicdataandanalysis.htm 

18 http://piaacgateway.com/datasets/ 

19 http://nces.ed.gov/train ing/datauser/#P!AAC 
20 http://piaacdataexplorer. oecd.org/ide/idepiaac/variables.aspx 
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There are four steps to progress through when using the PIAAC Data Explorer, supported by the 
DLDT modules. We outline our journey through the process as follows. 


Step 1 

In the first step we chose a Subject (Numeracy ) from three options: Literacy, Numeracy, and 
Problem Solving in Technology-rich Environments. Note that Adults (16-65) is the only option 
for Age at this point. After the choice of Numeracy, a fuller screen appears. Initially there is a 
choice of which Background Questionnaire dataset to use: PIAAC 2012, ALL 2003 or All 
years/studies', we chose PIAAC 2012. 

Next we selected a dependent variable from the 72 Categories and Groups available. Then we 
selected a jurisdiction or group from the 20 national and 17 sub-national entities available, as 
well as the OECD average. Under the Category Scale Scores, and Sub-Category Skills, the 
Measure PIAAC Numeric: Numeracy is already selected. Under the Group International, we 
chose the jurisdiction OECD average. We clicked on Select Variables to move to Step 2. 


Step 2 

At Step 2, we selected independent variables (from a choice of 334) in order to examine the 
strength of associations between these and the dependent variable chosen above. Under the 
Category (and Sub-Category) Major reporting group we chose the variable All Adults. In 
addition, under the Category International Background Questionnaire, and Sub-Category Skill 
use - literacy & numeracy, we chose the following six variables: 

• Skill use work - Numeracy - How often - Calculating costs or budgets 

• Skill use work - Numeracy - How often - Use or calculate fractions or percentages 

• Skill use work - Numeracy - How often - Use a calculator 

• Skill use work - Numeracy - How often - Prepare charts graphs or tables 

• Skill use work - Numeracy - How often - Use simple algebra or formulas 

• Skill use work - Numeracy - How often - Use advanced math or statistics 

We then clicked on Edit Reports to move to Step 3. 


Step 3 

Step 3 allowed us to choose the types of statistics we wanted to report, collapse any variable 
response categories, adjust table layouts and refine the formatting of the reports to be generated, 
using the variables chosen above. To summarize, we trialled the use of the PIAAC International 
Data Explorer (PIAAC IDE) 21 by exploring information about the percentage of adults currently 
or recently in work who self-report the frequency of their engagement in given numeracy 
practices in their work contexts, and their average Numeracy scale scores for each category. We 
focused on the PIAAC jurisdiction OECD average, then chose as variables Skill use work for 
each of the six specified numeracy skills in order to generate the average PIAAC Numeracy 
scale scores and the percentages of adults aged 16-65 in each usage response (frequency) 
category. 


21 http://piaacdataexplorer. oecd. org 
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The PIAAC International Data Explorer generated seven draft reports. These seven reports can 
be reviewed, edited, deleted or copied. New reports can be created, new formats and statistics 
chosen. We selected Averages and Percentages from the drop-down menu Statistics Options. 

We clicked on Build Reports to move to Step 4 to view the completed reports. 


Step 4 

At the fourth step PIAAC IDE generated a report for each of the six variables of Numeracy skill 
use at work. Each report lists Averages of the Numeracy scale scores and Percentages of the 
population (each with Standard Errors) for the response categories: Never; Less than once a 
month; Less than once a week hut at least once a month; At least once a week but not every day; 
Every day. Each report table can be viewed by selecting the report name from the drop-down 
menu. At this stage, we could preview and select information displayed in these tables or charts. 

In addition, PIAAC IDE will generate (on request) comparisons between frequencies of skill use 
with significance tests for each of the six variables ( Numeracy skill use at work ) for the statistic 
specified. This allowed us to compare, for example, the means of two frequency groups to see if 
the difference is statistically significant. For this trial run we requested these significance tests be 
generated for each numeracy skill for both percentages and averages. Our use of large samples 
will mean that almost all differences will be statistically significant. (This issue is discussed in 
the section above entitled “Analysing adult numeracy practices: issues and types of analyses”). 
Our purpose in this instance was to examine the tables that would be generated by PIAAC IDE. 

We then chose Export reports and downloaded our selected reports as Excel worksheets, 
allowing us to manipulate the data generated. Tables 1, 2 and 3, below, were produced through 
such a reorganisation of these Excel original reports/tables. At this stage we could have chosen 
instead to download the reports as HTML or Word documents. 

The first six reports, one for each of the six variables of Numeracy skill use at work, are 
summarised in Table 1 (below). 

Table 1 

Average Numeracy scale scores and percentage of adults for frequency options for Numeracy skill 
use at work variables 22 


Never 


Less than once a month 


Less than once a week but at At least once a week but not 
least once a month every day 


Every day 


Year/S ...... . Std Per- Std A Std Per- Std . Std Per- Std _ Std Per- Std „ Std Per- Std 

Jurisdiction Avge Avge Avge Avge Avge 

tudy error cent error error cent error error cent error error cent error error cent error 

Average of Numeracy scale scores and Percentage of adults by Frequency of Calculating costs or budgets in a work context 


2012 OECD Average 264 (0.3) 49 (0.2) 289 (0.6) 10 (0.1) 290 (0.7) 8 (0.1) 286 (0.5) 11 (0.1) 280 (0.4) 21 (0.1) 


Average of Numeracy scale scores and Percentage of adults by Frequency of Use, or calculations of, fractions or percentages in work context 
2012 OECD Average 257 (0.3) 46 (0.2) 283 (0.6) 9 (0.1) 291 (0.6) 8 (0.1) 293 (0.5) 13 (0.1) 290 (0.4) 23 (0.2) 


Average of Numeracy scale scores and Percentage of adults by Frequency of Use of a calculator in a work context 

2012 OECD Average 254 (0.4) 30 (0.2) 278 (0.7) 8 (0.1) 284 (0.7) 8 (0.1) 286 (0.5) 17 (0.1) 283 (0.3) 37 (0.2) 

Average of Numeracy scale scores and Percentage of adults by Frequency of Preparation of charts, graphs or tables in a work context 
2012 OECD Average 261 (0.3) 60 (0.2) 290 (0.5) 13 (0.1) 298 (0.5) 10 (0.1) 300 (0.6) 10 (0.1) 294 (0.8) 6 (0.1) 

Average of Numeracy scale scores and Percentage of adults by Frequency of Use of simple algebra or formulas in a work context 
2012 OECD Average 260 (0.3) 55 (0.2) 290 (0.6) 9 (0.1) 296 (0.7) 7 (0.1) 297 (0.6) 11 (0.1) 292 (0.5) 17 (0.1) 


Average of Numeracy scale scores and Percentage of adults by Frequency of Use of advanced math or statistics in a work context 
2012 OECD Average 270 (0.2) 86 (0.1) 303 (0.7) 7 (0.1) 306 (1.0) 3 (0.1) 305 (1.2) 2 (0.1) 297 (1.4) 2 (0.0) 

NOTE: The Numeracy scale ranges from 0 to 500. Some apparent differences between estimates may not be statistically significant. 

Average (Avge); Percentage (Percent); Standard Error (std error). 

SOURCE: Organization for Economic Cooperation and Development (OECD), 

Program for the International Assessment of Adult Competencies (PIAAC), 2012. 

Reports used for this table were generated using the PIAAC International Data Explorer.http://piaacdataexplorer.oecd.org 


“The data used were the OECD Average for PIAAC 2012. 
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Looking at percentages in Table 1 across the six Numeracy skill use at work variables (rather 
than within each skill), we found that the percentages of adults responding TVeverwith regard to 
the category, Numeracy skill use at work - Use a calculator was the lowest (30%) (of all the six 
numeracy skills) and the percentages of adults in the response category Every day was the 
highest (37%) (of all six skills). The use of calculators in the workplace therefore emerges as 
very common, since less than one third of adults in, or recently in, work indicate that they never 
use them at work. 

The next similar frequency response pattern is Use or calculate fractions or percentages with 
46% indicating Never and 23% indicating Every day, although Calculating costs or budgets is 
similar, with 49% and 21% respectively in these response categories. In other words, almost one 
half of adults in, or recently in, work reported that they never used these two numeracy skills at 
work. Focusing on averages across the six Numeracy skill use at work variables (rather than 
within each skill), we found that the averages of Numeracy scale scores within the frequency 
option Never are the lowest (of all frequency categories) for all six skills. These percentages and 
averages for the less complex Numeracy skills may indicate that adults with the lowest numeracy 
skill are choosing to work in jobs that entail infrequent numeracy activity. Equally it may 
indicate the converse - employers may recruit staff in a way that closely matches skills to the 
job, or infrequent numeracy practice may lead to skill loss. If the latter, this would support the 
findings of research on the British Cohort Studies that showed adults’ skills diminishing with 
lack of use (Bynner & Parsons, 1998, 2000). 

For the more complex numeracy skills, Numeracy skill use at work - Use advanced math or 
statistics shows percentages of adults in the response category Never as the highest (86%) (of all 
six skills) and percentages of adults in the response category Every day as the lowest (2%) (of all 
six skills). The next similar (but less striking) skill response pattern is Prepare charts graphs or 
tables, with 60% indicating Never and 6% indicating Every day, although Use simple algebra or 
formulas is similar, with 55% and 17% in these response categories respectively. Averages of 
numeracy scale scores for Numeracy skill use at work - Use advanced math or statistics are 
somewhat higher within most response categories. These percentages and averages may suggest 
that people with strong numeracy skills have jobs that entail frequent numeracy activity. 


Table 2 

Frequency of use of a calculator in a work context. Significance tests of the differences between either 
percentages or averages, calculated for each of the five frequency options 


Difference in percentages between variables Difference in averages between variables 



Never 

Less than 

once a 

month 

Less than 
once a week 
but at least 

once a 

month 

At least once 
a week but 
not every day 

Every day 



Never 

Less than 

once a 

month 

Less than 
once a week 
but at least 
once a month 

At least once 
a week but 
not every 
day 

Every day 

Never 


> 

Diff = 22 
P-value = 0 

> 

Diff = 21 
P-value = 0 

Diff = 13 
P-value = 0 

< 

Diff = -7 
P-value = 0 


Never 


< 

Diff = -24 
P-value = 0 

< 

Diff = -30 
P-value = 0 

< 

Diff = -32 
P-value = 0 

< 

Diff = -29 
P-value = 0 

Less than 
once a month 

< 

Diff = -22 
P-value = 0 


< 

Diff = -1 
P-value = 0 

Diff = -9 
P-value = 0 

< 

Diff = -29 
P-value = 0 


Less than 
once a month 

> 

Diff = 24 
P-value = 0 


< 

Diff = -6 
P-value = 0 

< 

Diff = -8 
P-value = 0 

< 

Diff = -5 
P-value = 0 

Less than 
once a week 
but at least 
once a month 

< 

Diff = -21 
P-value = 0 

> 

Diff = 1 
P-value = 0 


Diff = -9 
P-value = 0 

< 

Diff = -28 
P-value = 0 


Less than 
once a week 
but at least 
once a month 

> 

Diff = 30 
P-value = 0 

Diff = 6 
P-value = 0 


< 

Diff = -2 
P-value = 
0.0245 

X 

Diff = 1 
P-value = 
0.2616 

At least once 
a week but 
not every day 

< 

Diff = -13 
P-value = 0 

> 

Diff = 9 
P-value = 0 

> 

Diff = 9 
P-value = 0 


< 

Diff = -20 
P-value = 0 


At least once 
a week but 
not every day 

> 

Diff = 32 
P-value = 0 

Diff = 8 
P-value - 0 

> 

Diff = 2 
P-value = 
0.0245 


> 

Diff = 3 
P-value = 0 

Every day 

> 

Diff = 7 
P-value = 0 

> 

Diff = 29 
P-value = 0 

> 

Diff = 28 
P-value = 0 

> 

Diff = 20 
P-value = 0 



Every day 

> 

Diff = 29 
P-value = 0 

> 

Diff = 5 
P-value = 0 

Diff = -1 
P-value = 
0.2616 

Diff = -3 
P-value = 0 


LEGEND: 







< Significantly lower. 

> Significantly higher, 

x No significant difference. 

NOTE: Within country comparisons on any given year are dependent with an alpha level 
of 0.05. 

OECD Average, PIAAC 2012 

Organization for Economic Cooperation and Development (OECD) 

Program for the International Assessment of Adult Competencies (PIAAC) 

Generated using the PIAAC International Data Explorer. 
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Examples of the types of reports generated by PIAAC IDE in the form of tables of differences 
between percentages and averages, with associated p-values 23 are shown in Tables 2 and 3, 
below, for two numeracy skills (percentages and averages are combined). 

To see how one percentage (or average) compares with those for other frequencies, users should 
read across the row for that value in Tables 2 and 3. The displayed symbols indicate whether that 
value is significantly higher, significantly lower, or not significantly different from the value 
associated with that column.The p-value indicates the probability with which a difference in 
percentages (or averages) between frequency groups as large as observed here could occur by 
chance, if there were actually no difference. The customary significance level is 5% (0.05). The 
p-value must fall under this significance level for the results to be deemed statistically 
significant. As expected, most differences were statistically significant with very low p-values. 


Table 3 

Frequency ofpreparation of charts, graphs or tables in a work context. Significance tests of the 
differences between either percentages or averages, calculated for each of the five frequency options 


Difference in percentages between variables Difference in averages between variables 





Less than 






Less than 
once a week 
but at least 
once a month 

At least once 
a week but 
not every 
day 




Less than 

once a week 

At least once 




Less than 



Never 

once a 

month 

but at least 

once a 

month 

a week but 
not every day 

Every day 


Never 

once a 

month 

Every day 



> 

> 

> 

> 



< 

< 

< 

< 

Never 


Diff = 48 

Diff = 50 

Diff = 50 

Diff = 54 

Never 


Diff = -29 

Diff = -37 

Diff = -39 

Diff = -32 


P-value = 0 

P-value = 0 

P-value = 0 

P-value = 0 


P-value = 0 

P-value = 0 

P-value = 0 

P-value = 0 


< 


> 

> 

> 


> 


< 

< 

< 

Diff = -3 
P-value = 
0.0004 

Less than 
once a month 

Diff = -48 
P-value = 0 


Diff = 2 
P-value = 0 

Diff = 2 
P-value = 0 

Diff = 7 
P-value = 0 

Less than 
once a month 

Diff = 29 
P-value = 0 


Diff = -8 
P-value = 0 

Diff = -10 
P-value = 0 

Less than 

< 

< 


Diff = 0 
P-value = 
0.2277 

> 

Less than 

> 

> 


< 

Diff = -2 
P-value = 
0.01 

> 

once a week 

Diff = -50 

Diff = -2 


Diff = 4 

once a week 

Diff = 37 

Diff = 8 


Diff = 5 

but at least 

P-value = 0 

P-value = 0 


P-value = 0 

but at least 

P-value = 0 

P-value = 0 


P-value = 0 

once a month 





once a month 





At least once 
a week but 
not every day 

< 

Diff = -50 

< 

Diff = -2 

Diff = 0 
P-value = 
0.2277 


> 

Diff = 4 

At least once 
a week but 
not every day 

> 

Diff = 39 

> 

Diff = 10 

> 

Diff = 2 


> 

Diff = 7 

P-value = 0 

P-value = 0 


P-value = 0 

P-value = 0 

P-value = 0 

P-value = 0.01 


P-value = 0 


< 

< 

< 

< 



> 

Diff = 3 
P-value = 
0.0004 

< 

< 


Every day 

Diff = -54 
P-value = 0 

Diff = -7 
P-value = 0 

Diff = -4 
P-value = 0 

Diff = -4 
P-value = 0 


Every day 

Diff = 32 
P-value = 0 

Diff = -5 
P-value = 0 

Diff = -7 
P-value = 0 



LEGEND: 


< Significantly lower. OECD Average, PIAAC 2012 

> Significantly higher. Organization for Economic Cooperation and Development (OECD) 

x No significant difference. Program for the International Assessment of Adult Competencies (PIAAC) 

NOTE: Within country comparisons on any given year are dependent with an 

alpha level of 0.05. Generated using the PIAAC International Data Explorer. 


Conclusion 

Overall, we found our exploration of the publicly-available PIAAC dataset stimulating and 
challenging in equal measure. While the scale of the dataset may appear daunting to novice 
researchers and practitioners, we would encourage readers to undertake their own exploration, 
using the range of support and tools for analysis available online. Our focus here has been on 
adults’ numeracy practices: just one of the many areas of interest on which data are available in 
this rich resource. Ultimately, the choice of focus lies with the reader. Our advice is: start with a 
simple question - something that intrigues you - and take it from there. 


23 See below for an explanation of p-value. 
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